Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

/ok to test

PR 2150 first CI failure analysis

Workflow: https://github.com/NVIDIA/cuda-python/actions/runs/26591678170

Commit: 293258d

Workflow result: failed.

High-level result

The build and non-test infrastructure mostly passed:

• Build jobs passed.
• Docs passed.
• pre-commit.ci passed.
• The final Check job status job failed because matrix test jobs failed.

The failures are concentrated in test matrix jobs. There were 37 failed test jobs plus the final status aggregation job.

Failure counts by CUDA version:

• CUDA 13.3.0: 24 failed test jobs.
• CUDA 12.9.1: 13 failed test jobs.
• CUDA 13.0.2: no failures observed in the failed-job list.

Failure counts by platform:

• linux-64: 17 failed test jobs.
• linux-aarch64: 12 failed test jobs.
• win-64: 8 failed test jobs.

Failure mode 1: CUDA 13.3 missing CUcheckpointRestoreArgs

This is the expected proof-of-coverage failure for the issue fixed by PR #2144.

Representative failed jobs:

• https://github.com/NVIDIA/cuda-python/actions/runs/26591678170/job/78353772395
• https://github.com/NVIDIA/cuda-python/actions/runs/26591678170/job/78354692898

Observed failure:

tests/test_cuda.py::test_cuCheckpoint_required_bindings_present FAILED
E       AssertionError: assert ['CUcheckpointRestoreArgs'] == []
E         Left contains one more item: 'CUcheckpointRestoreArgs'

This shows the new cuda_bindings/tests/test_cuda.py::test_cuCheckpoint_required_bindings_present test catches the missing generated binding directly. This is exactly the failure PR #2150 was intended to expose before PR #2144 is merged/rebased in.

Failure mode 2: Linux CUDA 12.9 missing CUcheckpointGpuPair

This is a separate checkpoint binding gap surfaced by tightening the cuda.core checkpoint availability guard.

Representative failed jobs:

• https://github.com/NVIDIA/cuda-python/actions/runs/26591678170/job/78353772318
• https://github.com/NVIDIA/cuda-python/actions/runs/26591678170/job/78354682523

Observed failure:

ERROR collecting tests/test_checkpoint.py
RuntimeError: CUDA checkpointing requires cuda.bindings with CUDA checkpoint API support. Missing: CUcheckpointGpuPair
binding_ver = (12, 9, 7)
missing    = ['CUcheckpointGpuPair']

This happens during cuda_core/tests/test_checkpoint.py collection on Linux. Windows does not hit this path because the checkpoint tests are platform-skipped before _checkpoint_available() is evaluated.

This is not the CUDA 13.3 CUcheckpointRestoreArgs issue. It means the tightened guard also exposes that CUDA 12.9 Linux bindings report enough checkpoint API surface to reach cuda.core.checkpoint._get_driver(), but still lack CUcheckpointGpuPair, which cuda.core considers required.

Interpretation

CI behaved as intended for the main goal: PR #2150 converts the previous skip-masked CUDA 13.3 binding regression into clear test failures.

The run also identifies a follow-up decision for CUDA 12.9:

• If CUDA 12.9 should support the cuda.core.checkpoint mapping helpers, then CUcheckpointGpuPair needs to be present in the CUDA 12.9 bindings.
• If CUDA 12.9 should not support that surface, the cuda.core availability check needs to classify this specific older-binding condition as skippable rather than as a binding regression.

After PR #2144 is merged and PR #2150 is rebased onto it, the CUDA 13.3 CUcheckpointRestoreArgs failures should disappear. The CUDA 12.9 CUcheckpointGpuPair failures may remain unless they are handled separately.

I looked into the CUDA 12.9 failures from the first PR #2150 CI run.

The short version: these failures look separate from the CUDA 13.3 CUcheckpointRestoreArgs regression that PR #2144 fixes.

In /usr/local/cuda-12.9, I do not see CUcheckpointGpuPair at all. Even

grep -r -i GpuPair /usr/local/cuda-12.9

returns no matches.

The CUDA 12.9 cuda.h checkpoint restore args are still the older reserved-only layout:

typedef struct CUcheckpointRestoreArgs_st {
    cuuint64_t reserved[8]; /**< Reserved for future use, must be zeroed */
} CUcheckpointRestoreArgs;

That matches the CUDA 12.9 CI failure mode from https://github.com/NVIDIA/cuda-python/actions/runs/26591678170: Linux CUDA 12.9 jobs now fail during cuda_core/tests/test_checkpoint.py collection with:

RuntimeError: CUDA checkpointing requires cuda.bindings with CUDA checkpoint API support. Missing: CUcheckpointGpuPair
binding_ver = (12, 9, 7)
missing    = ['CUcheckpointGpuPair']

So my current interpretation is:

• The CUDA 13.3 failures are expected and useful: PR Make checkpoint tests fail on missing required binding symbols #2150 proves that missing CUcheckpointRestoreArgs would now fail loudly instead of being skip-masked.
• The CUDA 12.9 failures are a separate compatibility issue surfaced by the tighter guard.
• Since CUcheckpointGpuPair does not appear to exist in the CUDA 12.9 headers, this is not a missing 12.9 Python binding. It is more likely that cuda.core.checkpoint is treating CUcheckpointGpuPair as required too broadly for CUDA 12.9.

Possible follow-up direction: keep missing required symbols as failures for APIs that should exist in the active CUDA version, but treat the CUDA 12.9/no-CUcheckpointGpuPair path as an older checkpoint API shape that should remain skippable or should avoid enabling the GPU remapping surface.

/ok to test

PR 2150 second CI failure analysis

Workflow: https://github.com/NVIDIA/cuda-python/actions/runs/26596635176

Commit: cd730c1

Current workflow state at inspection time:

• 65 jobs passed.
• 23 jobs failed.
• 1 job was still queued: Test win-64 / Python 3.14, CUDA 13.3.0 (wheels), GPU h100 (x2) (MCDM).

High-level result

The second CI run matches expectations after splitting baseline checkpoint support from GPU-remapping support.

All completed failures are CUDA 13.3.0 test jobs. CUDA 12.9.1 and CUDA 13.0.2 jobs that completed are passing.

Failure counts by CUDA version:

• CUDA 13.3.0: 23 failed test jobs.
• CUDA 12.9.1: 0 failed test jobs observed.
• CUDA 13.0.2: 0 failed test jobs observed.

Failure counts by platform:

• linux-64: 10 failed test jobs, all CUDA 13.3.0.
• linux-aarch64: 6 failed test jobs, all CUDA 13.3.0.
• win-64: 7 failed test jobs, all CUDA 13.3.0.

Remaining failure mode: CUDA 13.3 missing CUcheckpointRestoreArgs

Representative failed jobs:

• https://github.com/NVIDIA/cuda-python/actions/runs/26596635176/job/78371061954
• https://github.com/NVIDIA/cuda-python/actions/runs/26596635176/job/78372022514

Both sampled logs show the same intended failure:

tests/test_cuda.py::test_cuCheckpoint_required_bindings_present FAILED
E       AssertionError: assert ['CUcheckpointRestoreArgs'] == []
E         Left contains one more item: 'CUcheckpointRestoreArgs'
missing    = ['CUcheckpointRestoreArgs']

This is the expected pre-PR-2144 signal. PR #2150 is still proving that the CUDA 13.3 CUcheckpointRestoreArgs binding regression would fail loudly instead of being skip-masked.

Fixed compared with the first CI run

The CUDA 12.9 CUcheckpointGpuPair failure mode is gone.

In the first run, Linux CUDA 12.9 jobs failed during cuda_core/tests/test_checkpoint.py collection with:

RuntimeError: CUDA checkpointing requires cuda.bindings with CUDA checkpoint API support. Missing: CUcheckpointGpuPair
binding_ver = (12, 9, 7)
missing    = ['CUcheckpointGpuPair']

After commit cd730c1, completed CUDA 12.9.1 jobs are passing. This confirms that baseline checkpoint tests no longer require the CUDA 13.x GPU-remapping type, while the CUDA 13.3 missing-CUcheckpointRestoreArgs regression remains a hard failure.

Interpretation

The results are the desired intermediate state:

• PR 2150 no longer over-requires CUcheckpointGpuPair for CUDA 12.9.
• PR 2150 still catches the original CUDA 13.3 CUcheckpointRestoreArgs generation issue.
• After PR 2144 lands and PR 2150 is rebased onto it, the CUDA 13.3 failures should disappear.

Thanks Leo, fair concern on maintainability.

This was AI-assisted, but I manually guided and reviewed it. The intent is not to build a broad ad hoc API coverage framework in this PR. The new cuda_bindings test is a deliberately narrow regression guard for the exact checkpoint binding surface that silently disappeared in 13.3.0. Keith also raised the broader version of this in Slack: we should have something that enumerates all public APIs and types per CUDA version so parser failures cannot silently drop symbols. I agree with that, but it is a much larger ask.

For cuda_core, the goal is also narrower than adding new checkpoint scenario coverage. The existing checkpoint tests already covered the restore path, but the availability helper treated missing required bindings as an unsupported-environment skip. This PR fixes that boundary: unsupported drivers/old bindings still skip, but missing required symbols now fail. The added cuda_core tests are focused on that skip/fail behavior and on separating baseline checkpoint support from GPU remapping support.